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ABSTBACT * ' . ' 

Tbis study was undertaken to deteriine iifaetber 
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psychology , Social psychology, I^ysiolcgica 1 and Conparativ^ 
psychology. Perceptual and Sensory psycbolcgy^. and Clinical and 
AbnordTal psycbclogy) were identified tj the 6B£ Advanced Psychclogy 
Test COnnittee of Exaniners. These experiiental £ub£Cores# the tvo 
currently ^.ported subscores, and the tctdl sccre Her^ acalyzed« 
An^lly^is sbOHcdi thaty for »ost studentjs# additicnal infonation about 
Strengths and ireakn«s<s in sc8€ of the areas ccilld be ottain«d. The 
particular subscores vhich could provide useful infcriaticn varied 
fro^ stta^dent to^ student. This finding .was supported by an exaninatlon 
of fifty xrandofily chosen ansiter sheets. It itas. concluded that 
subscores^based on the content areas identified by the Psychology 
CoBsittee nay^ave potential for providing additional information for 
'puri)oses of guidance and placeiaeiit. Suhscqres 'kased cn a -^i act or 
afialyj^is of the test, hoHever, were judged not to have equivalent 
potential. -<AuthOr/ECF) ' 
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ABSTRACT 

This study va^ undertaken to determine whether additional 
information useful for guidance or placement could be derived- from 

existing Advanced Tests. The number of subscores currentrly reported 

^ ^ ' ^ \ . ^ ^ ' 

for eacii Advanced Test*ls limited by the high reliability required 

J' 

for^ subscores used In making admissions decisions. S^ubscores used 
onl^ for guidance and placement vould not ne^d'tor me^t such a 
^rigorous standard of reliability. Subficorea based eigh^ content 

areas were Identified, by the GRE Advanced Psychology Te^t Commlt^tee 

^ > ,1 

of Examiners; analyses of these experlmental^subscoires/of the tvo 
currently reported subscor^s^ and of the total scdre ^ere (^p^rled 
out for two forms of the Advanced Psycholosy Test. 

>• ■ ■ ■ I, ■ . 

Analysis of the reliability of the idlf fereiipes among the 
exper;UentaJ. subscores showed that^ foremost studentaj additional 
Info'rmatlon about strengths and weaknesses In saom^ of the eight 
subscore areas could be obtain^. The particular subscores for^ 
"which useful lnfbrmatlon*could*be obtained varied from student to 
student. Thl^ finding was' supported by an examination of fifty 
randomly chosen answer sheets. It .was concluded tha^ spbscores , 

based on the cotitent areas Identified by the Committee; of Examiners 

1 * * " 

may have potential for /providing additional Informati^on for purposes 
of guidance and placement about mbst: students* who t^ke the Advanced 
Psychology Test.' Subscores abased on a fatS'tor analysis of the test^ 
^however^ were ludged not to *have equivalent potential 



.An Invastisacion of the Feasibility of 
Obcaittlng Additional- Subscores on the 
,GkE Advanced Psychology Test^ ' 

GRE No, 74^ , . \ 



In October 1972 the GRE Program began reporting subscores for a 
H'jiiber of irs Advanced Tests, For several years prior 'to that^d^ca. 



subscor^ .^ad been the subject of- uldespraad discissions, and a number 
of the G3t£ Coamittees of ^Eiaainers responded positively to the GR2 
Board*s :*ecoTaaendation that, subscores be reported* The decision to 
reporr s'^bscores'scenased from a feeling on the part of, both the Board 
ard several of th^e Comictei&s of Exaiainers that more inf ortnatipn shou^ld 
be produced from three hours of testiig tliaa a single score and that ' 
the. tests vould be taore us^ul if th&y could indicate strengths and 
weaknesses la the*saveral subfields of each^ content area* ^^^^ 
Board £i:4 Committees recogni2ed that subscores vould be valtiable for 
isariy counseling and placement decisions^.^ ' * 

In spice of the widespread agreemenc about the d4sirabiiity of* 
reporting subscores, there were a number of areas of coacerri — ^ includ- 
ing the reliability and independence of the subscores — which ^entered 
on tj^ef use to' which the subscores would eventually be put. If the use 
of subscores can be restricted to placement and guidance — primarily 
through the tining of reporting of subscores — the importance of these 
toacems vill dininish* Placement and counseling decisions are reversible, 
whereas a^^aissions decisions generally ^re not; therefore, luCtch lower 
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j^jjj^dards of statistical adequacy ^ould be applicable .if subscor^ ^^ere 
not ^us^' for admissions decisions: 

,Bas^ on then current GRE Program reporting practices — an^ phe 
feair fi^t test Tj^ers would fail to' heed advice given with regard to the propet 
use^of g^scores — a decision was r^ched to limit the it^itial develop." 
;;sat ol sufescores to those wh^ch attain a reliability of at -least *3b 
and which ha%& a/'corrected" i^tercorrelation with other subscores o£ 

less than .90; \Based'on these decisions, no laore than four such sub- 

\ 

scores could be o'^ained from any of Che Advanced Tests, and each of the 
subscores would have, to be ba^d on approximately SO to 80 items In order 
'to obtain the required reliability. Faced with this limitatioii, serveiral • 
Coiriittees werfe reluc(;ant to tailor their test specif ications' to laeet 
the statistical requirements (particularly the need for SO i;o 80 iteiis 
for each subscore)^ opting instead for coverage ^of ^he elements in their 
discipline in a manner more consistent with emphasis in the undergraduate 
curriculum* Other ComJ^t^ee s, ^in clud ing the Psychology Committee, adopted 
subscores under the limitations described above. 

. * Since the introduction of subscoxes in operational tests in 1972, 

bdth the GRZ Bqard Research Committee and the Psychology Committee 

t * 

hava^ continued to express an interest in investigating the possibility 
o5^ reporting more subscores, particularly for guidance purposes- At its 
cAeting in Hay, 1973, an^i-^gain in May 1974, the Psychology Conanittee 
l:!Ldjtcated a desire to experiment with the creation of subscores based 
upon the^contenf area specifications naw in use^ While recognizing the 
^Tosential ^reaknesses in subscores developed ^in tKls manner, tlife Cot3iittee 

coneet.eless *f alt that the potential value of chfesa additional subscores 

' ^ ^ ■ 

for cpunsaliag and placem^t purposes wafranted additional investigation 

* 

C • ■ \ 
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of 30th rlla possibility of developing such s^bscores and the reporting 
:r£:;hod5 ^hat vould ba necessary- to mip±3l2e their potential inisuse,- - 
Taasa fealinjs vare reiaforcey ia discussions during me&tiags of 
tha G5S Board Research Coccaittee, *which,was then involved in a review 
of t^.a, current and"^u£ure directions of the GR£ Program and of its 
rsleted activities* Beginning in April 197A and continuing at a special 
^*::^eting in Jun*, ths Rasearch'Committea considered both the broad 
direccioas in vhich the GRE Program might develop end ^ ntuaber of spe* 

x * 

clfic propositions concerning the future of the GRE. One of t^ese propo- 
sitions — endorsed at th^*June 1974 meeting df the Research Committee 
and at the Sapcenbar 1974 meeting of the full GRE Board — stated: '-^ 

* 

"Ic is both desirable an<l feasible to report tadr© detailed and,' 
usef'-il Dart**score information on the^ basis of the Advanced Tests* 

fa, This expansion could provide for ther generation of additional 
' * * ' ' ' •* 

suoscores for ell tests- in the current format* - * 

A ^ ^ ^ 

b- This^expension could also provide for tha gene^raCion^of sub- * 

scores deisigaed for use in guidance and placement*" ^ 



II. Purpose end Procedures * * ^ \ 

As e result of these discussions, the current study was designed 
10 investigate the number of logically meaningful subscores that can be 
ganeratad from the Advanced Psychology Test without be\ng bound' before- 
hand by a prespecified reliability which the subscbres must; >att3tiri* 
Presioiag chat* the subscores would be used only for guldancce ^atj^ place- 
cent purposes and not as an integral part of the admissiOT^ process, 
The statistical standards for reliability coulS be greatlj^ reduced. 
This relaxation would, enable many aore suisscores to .^e reported, tAile 



' 1 * 



allowlagf tesc conmltce'es to give continued emphasis to ths various 

elenents Xn their tiisciplin^. Thus, this scudy eicsi^lned the raliabtliny 

' - * 

and Ind-epandance of subscores based on the eight major coateac areas of 
tha 'Advanced Psychology Test as Initial step in determining the extent 
Co which tha propositions endorsed' by the GRE Board were conceptually 
and p5ych<i3etrically feasible for expansion to the design and administra- 
tion of the, ORE Advanced Tests, " ■ .-"^ ' , 

Content analysis was used to 'define the structure of two forms , 
^CUGR2 and VGRl) of Che GRE Advanced Psychology Test based on the assui^p- 
tion that a subscore might be developed in each of tne n^ne content areas 
established by the Cotnsirittee 'aaj^art of its test specifications; in ad- 
dition, a factor analysis was performed to obtain a less subjective exam- 
lotion of the structure of the tests. The analysis was based on tvo 
samples- selected from the October administration of the ORB; score 

distributions were computed and compared to stmilar^ score distributions 
for the Advanced IJ^ycholog;^ Test over the previous three-year period and 
were judged to be essentially the same. As part of the process of con* 
ducting the content and factor analyses (described la more detail' in the 
following sections), both prelininary and final. results were presented to 
and discussed with the Psychology Committee in March of 1975 and January 
of i;9J6, respectively- Finally, conclusions were drawn, which are presented 
in the final section of this report , ' , - 

III. The cSntent Analysis 

As noted above, the current test a(^eclf ications for the Advanced 
Psychology" Test Include'nine Content areas; these'areas and the number 
of questions associated with each of the areas in each of the two forms 
of the test are presented In Table 1 on the following pagej * . 



8 




Table 1^ • . 



Content Area * 


s 

Nuabet of 
Quesmons ^ 


i • 


UGR2 


\ VGRl 


Personality , 


24 


25 


Learning* ^ 


, JO 
^ 


35 


Mea^ure^ient 


25 


25 


Develop2:ental Psychology 


15 


15 


Social Jsycholggy 


23 


20 


' Physiological and Cocpatative Psycholo^gy 


25 


25 ; 


Perception and Ser.3ory Psychology 


30 


30 


Clinical and Abnor^ial Psychology 

— — ■ 


17 


20 


*Ap?dLied Psyc'^iology and Other 


' 5 


5 



As a first ?tep, routine item analysis and test analysis were per-, 
fgrtied on the^ two test forzis; the analysis included distributions of 
the subscores, ceaas, standard deviations, reliabilities, standard erro^rs 
oi measurazent, 'and a__correlatlon aatrix of the nine expsriaental sub-* 



* It is readily apparent th^t in thase'two forms of the test there are 
tJo fetr questions in the area of applied psychology to permit a viable 
subacore to'be obtained, a reflection of the fact that the -Psychology ^ 
Cozaittee has felt that this area is not important enough in the under- 
graduate curriculum to varrant greater coverage and had not recotnaended 
that an experimental subscore in that area be reported- Nevertheless, 
an analysis of that area was performed -So 1 that all the itens wuld be 
included in the analysis. 
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scor23 listad in Table i togeChe^wlCh tha Chree cutrenCly raporwed 



3138x1^1 correlations ^/itti aach of tha c:;alve scores for aach 



irea w&ra cocpuced. These corralawioas w*ere examined for evidanca thac^ 
so:::a of the items laight be oisclassif iad . A few iceras were discdvere<i 
vhich had a higher cbrrelation vith a subscore othar than the oae into 



;av vere placed by tha Committee; in each insCance,, however 



conTanC o: cr^e iten clearly fell vithin tha area of cfie subscare fa which 
it had baan placed, Tablas suisaarizing these results are presented 

Tna corralacions for pairs of subscores, corrected for aCtenuation, 
ire pressrtted in Table 2 on the following pagei (correlations prior to 
corrsction ara presented in A^endiz II) , ■ Table 2a concains the correla- 
risns cczDLited* from the VQR2 data -and Table 2b dontains the corralations 



:::?utad f ron xthe VGRl data. These correlations tend to confirm th* 



;:-ita= 



of Exaiainers* xeeliag th^'t 4ach of the eKperisental subscorss 



contains souie unique Inforcanlon, 

It is apparent froii^ Table 2 that some of tha pairs of e3Cper^J^antal 
si;b$cores> for .instance subscores 1 a^d 6 an<J subscores 5 aad 6, ha-^e. - 
=uch lovar cor'^elations wich each other than 'do the subscores being 
raporcad at present. Only two entries in the tabla exceed the ,^90 
linit set for the intercorrelation of subscores* In each of these cases^ 
tha correlation computed fr*om the^data on the dther fom^of^the test 



* Zha thraa currently reported scores are a total score (based on all 
ite=i3), an a^eriaejital psyc?iolog7 sybscore (based on the items in 

» experir;ental subscores 2,^6, and'7), and a social psychology subscora 
erased on the items in e::p^i2iental subscores Ij.^j 5, and 8.) The 

* radian correlation,- correcte^J for attanyation, betvean these nvo^ 
reported subscores for the five :30st recent forns of the test is ,82*- 



10 



-7-; 



< 



- - 22. Cc*r^relEtior*s frosi UCR2 



! : i — ^1 

• .MM 


3 .! « .| 


1 ' ■ 


7 1 


■ 8 1 


1. Personality ] .75 j .S$ j ,?3 


.95 [ .58' 


.71 1 

^ 1 


1 


2, Learning i .7"$ 

* * , r 




.81 


..7^ ! .-61' 


,1 


.76 r 


3, .Measurement * ^ ; ! ,S3 | j -7^^ 


.84 1 .57 


.73 1 


I 


4. Developmental j j ,3i 


.74 




.83 


. 63 


.72 


— ^ r 

.83 ' 

i 


5. Social - i -53 ^ .75 


:s4 


*S3. 




. .54 


.72 . 




6.^ Physiological & Comparative . j ,5$ ! ,61 




. .63 


.54M... 


.75 


.64- 


7, Perceptual Sens^>ry , i 71 1 81 


■ .78 


, .72 


,72 [ .75 




.70 


8** t^L'lDical & Abnormal- ' ! .33 | -76 


.84 


.63 


.19 \ .64 


*.70 


t 


2b. Correlations frorn VGRl 


^ . .. ^ 




5 


6 


. 7 


8- 


1. Personality j* 


. .77 j .76 


,.74 


.68 


. .62 

I 


.67 




2 * Learning 


.77 

i 




.80 


.79 


.75 


.83 


.83 




3, Measurement - 


.76 ■ 


.80 




.82 . 


.74 






.75 


.73- 


4* D^veloptaental ■ . 74 


.79 1 .82 




.84 




\n 

* 


i 


i\ Social ^ ' ^ r . 68 


.75- 




.84- 




.65 


.72 




6, Physiological & Comparative I ^62 


.83 ' .77' 

■ 


.77 


.&5 

• 


1 ' 

1 


.84 




i i ■ 

7. Perceptual Sensory. 7 \ .67 

c ■ 1 . 


.83 ; .75 


.78 


.72 


.84 




.60 1 


8^ Glinical & Abnormal j .-90''' 


,76 i .73 

i 


.76 


.72> 


' .67 


^ .60 


L-.1 



. II. 
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for ths same two subscolres 'is substantially below .90. Thus, each of 
t'cet eigtvt sub^cqi^s appears to neet the criteria of independence set 
for ^bscores^in the GRE Program • -This result confirms the validity 
.of the Coramittee b£ Examiners' belief that subscores based on thS eight 
« Content areas would be about as independent as the two subscores being 
reported at present. The Cooait^ee feels that . subscores balfed on the 
^content areas would be b^-far the most useful for purposes of guidance and 
. placeasntj becaruse the curricuL-,;^ tends to be orgacnised in tlie same way. 
"'^ Tae reliabilities of che subscOtes based pn the eight ji^or contend 
areas were" compute^ from K\:dar-Kic^r^son formtila*'20 and^re given in 
T^ble 3 bela^.. The reliabilities depend in large measure on the number of 
items contributing to each subscdre. Subscores 2, Learhltig, and 7, 

ParceDtual h Sensory, have the highest reliabilities atfd are based^ .on ' 

* ^ CI 

< the most questions, 30 and 35 respectively. ^^Subscore A, Dsvelopmental 
Psychology, has the lowest reliability, and is bSsed csn"^ the fewest- number 
of questions, 15. The remaining Subscores are based oa 20 to 25 questions* 



Table 3 ^ 
Reliamlities of Subscores*^ 



* > 


TesttForm 


.Content Area 


UGR2 


VGRl 


_ 1. Personality 


.53 . 


.66 


2. Learning 


."80 




, '3, Measurement 




' .63 ^ 


4. , Developmental 


U5 




— '■ — ' ' m • ' ' 

5. ■ Social , ■ ^^^^ 


,£2 


^'57 


§1 Physito log leal >S Comparative \ .69 


.73 


7. Perceptual' ^( Sensory 

^ ■ ' ^ 


.78 


■.■77 • 


8. .Clinical & Abnorjial ^ 
a^.<>it^ ^— — ■ 


1 .53 


' .66 

> 



12 
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^ ^, Ica reliabt-lttj^f/thasa sub^coras is not iapressive when judged 
by^u^uel standards. 'Oaly the Learntftg* sufascore in VG^ reaches 'tKa 
s^aadard of re;Liability required of the. atibscoras being reported at 

*?ra3ant. liavertheless, ^thase' riel'iabiliales are respectable for subscores 
basad ozl 15 to 35 ques^toas. The reliaHility of the difference between 
any tvo of these sufascores wilj ^be , fairly mod&st . The question to be. ^ 
addressed is whether the reliability of the differences between these 
subscores is"hig€ enough to make them helpful in guidance ^nd placement 

Lord (1958) has ^ggestad one way of e^aminlitg this question. He 
suggests that the difference between any two test scores may be quite use- 
ful even. when this difference ha^ very low reliability by conventional 
standards,. ,He suggests that a criterion for the usefulness of difference . 
scores be set in^ terms of the proportion of students about whom judgments 
that one score is higher fihan another score can be made with 80 to 90 
percent accuracy. Ifi the context^of guid£Utce and^placpment^.the ability -4 
to dif f erentiate'among a 3tudent*s attainment in several subscore areas 
and to he right 80' to 90 percent of the time might well be worthwhile. 

Tae Usefulness f pr individual guidance of^^gy measure of lov relia- 
bllity is greatly increased uxider tWd conditions of use: 

1. The measure is used to make broad rather than praise 

w 

classifications. * ^ . 

2, Judgnents are/oadle only about those students with somewhat , 
\ extreme scores; judgment is made about those students 

with less extreme scores, 
"nesa conditions ere approxinated when difference scores are used in the 
. f ol^wing na n aer . 

. • - .. ' . - " 13 ' ; 
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^1, The subtest scores ere slocced on a profile chart as a'conct- 
deace band extendj^ng one standard error of* neasuteEie^it ebove / 
■ aod'belotf th,e obtained score, ' ' V.' 

• 2, The difference b&jtweeii scores on two_ subtests is" treated as 'V' 
real only when the two confidence bands do not overlap, 

■> * 

The effectiveness of ,atiy ^iifference scare when used in ,this way can 
be evaluated in term of the proportion of students for whom judgments 
ate made and the prop.ortion of correct judgments; some illustrative re- 
sults appear in table 4 below. . ^ * * . 

' Tfble 4* ^ ^ 

Proportion of Comparisons about which Judgments are Made * 
that Ther^' is a R^aal Difference Between Two Test Scores 



\ Reliabilic/ 


Cuccir.g' 
CO 


Proportion of ♦ 
Studencs for 
v-Tnon Judgments " 
Are ^fade 


7 

. PrVoortion . 

Of Correct 
1 Jud^nents 


; ^ .09 


1,35 ] 


.18 




1 ^ ',2925 \ 1.26 


>21 




\ .3025" . I' ''1.18 




^1 

.86 


,^ ,4223 « _ j , 1.07 






1.01 




.92 


1 '^v 


.85 


.40 


. 95 



This table .shows that difference scores with very low reliability can 
provide useful information^ Difference scores tflth' a reliability or only 



Lord, E^*M. , The Utilization of Unreliable -Difference Scores , Educational 
Testing Service, R3-5S-1, 19S8. (For a more 'extended discussion -ijc 
Lord*s work, see Appendix III-) ^ 



.20> for example^ can provide judgments .that there ts a real'difference'be-' 
tween two test scores^ for about a flrfth of the comparisons madq» with 80 / 
percent confidence in the wr^ectness of the Judgments made.' ,/^^> -^since' 
vthere ate twenty-eight comparisons be made among the ^eight^su^score$ for 



ea^trvtudentf^^ it is noti surprisirj^tha^ an* exaibination' pf^^ift^r randomly 
chosen answer jsheetfa^^-sh^Wed jfet^ a^^^ttional itiforma'tion could "^e repOJTted 
for the grejt^jffi/ity^'of J^^^i^'^ ,.t* ' . . \ ' 



'The areliabilii^^'Of the' dif£ei*ence''between ,eich- pair o£ ^u.bsoores' f or 
both test forms 



cms is given in TabJ^e 5 on the f o^^orving page. Of the 56 ve%ues^' 
given in table 5> A8 ^6C^rep[tef than .20» and 35 are greater tttan .30> 
This shdw^ that In the gr^at majority of comparison? between pairs of ^ut)- 
scores^ judgments could be made about 20 to 25 percent of the comparlsopSsf^ 
'ulth '80 to 85 percent accuracy. As noted above» given twenty-height com*- 
parisons per student^ the subscores a|)pear to have a substantial potential 
to provide useful information for purposes of guidance and placement. 

V In addition^ it should be noted that of the eight instances in which 
the stan4ard error of the difference between two subscores failed to reach. 
.20> five of them involved a comparison with' subscore 4^ Developmental 
Psychology^ which was bad^d on only 15 items and has a reliability of le^s ^ 
than .50 in each test for^. In more £^ec^nt.f orm$ of the test the Committee ^. 
has raised the number of items in this contenl^ 6'aCegory frojn^i^S^ CiS 20i 



Kinor changes of this kind in the test "specifications^ plus^^ slight jin-^ 
ctease in the total number of item^/the test is quite u^iv&ededgiy ^ould 



^erve to increase slightly the number of students al>outv^9m Judginents 



cdu4^be made ulth reasonable accuracy. ^ ^ 

tu summary^ it appeai:^ that stibsc/^res based on ^e^content areas de^ 
fined by the fiommitl^ee have considerable potential to provide Informatioiu 



Table 5 



. ■ ' ^ • ' , Reliability of the Difference 

Between Each of the Experiment at" Subscores 

* 5a» Data from'UGR2 





1 


' 2 


3 


4"' 

* 


1 


1 

- 


T 

1 


8 '* 


1* ' ?ersonatity 




.34 


.20 . 


.11 


! 

.06 


.40 


.38 


.17 


2» .beaming 


'.34 




.'29' 


' .27^ 


.34 


..S3 


.43 


■ .35. 


3* '>taasjLirenient ' <20 


.29 




.30 


.26 


.52 




.24 


' 4*^ -^DeveJ-^pdlental ^ j .11 


- .-27 


.30 




.17- 


■ .34 


.32 


. 13 


Social . 1 .06 


.34 


.26 


. .17 




.47 


.40 


.23 


6/ Physiolq^ical & Comparative j •AO 


.53 


.52- 


.34 


.47 




.AO 


.38 


7,^ Perc^tual & Sen^ory^* j .38 


.43- 


.42 




.40 


.40 




,39 


.'"3. cfinical^.,Abtioroa^- ' j .I7J .35 ~ 


.24 


.15 


.23. 


.38 


.39 




' , ' '5b. Data from VGRl 






•*2 


3' 


- 4 


5 


i 

6 


7 


8 


--P- ' — 

\ 1* Personald^ 




.37 


.'31. 


.26 


.34 


.46 


.46 


.17 


j - 2/ ' Learning 


.37, 




.32 


.28 




34 


.35 


.37 


.38 


^3, .Me&surenenc: 


.31 1 .32 

t 




.19 


.28' 


,34 




.33 


! 4*^ 'D^velopiftentai 


.26 


.28 


.19 




.15. 




. ,25 . 


.25 


' : ^ : — 

; 5, Social 


.34 


. .34' 


.28 


.15 




' .25 


.37 


■ .31 


\ 

\ 6. Pliysio logical & Comparative 


.46 


.35 


.34 


.28 


.25* 




.33 


-.44 


!• ' 7, Perceptual &,^ecisory 


.46 i 


-.37 


.37 


.29 


.37 


.33 




•51 


! 8. ^Clinical fit .Abnormal 

' (? 


.17 


'.38 


.33 


.25 , 


.31 


.44 


.51 


1 



*^out students with unusually hi|h low scbres for use la guidance and 
*plVcemep(^vj standard error ^pf thA>diff erence between the great m^* 
Jotity of pairs of subscores would allw judgments to be made about 20^to 
?5 -B^erc^ul: of ^the comparisons Vith 80 to 83 percent accuracy; a random 

5aiB(>lc^o£ attswer sheets shows that this would provide useful lofonnation 

* * ■ 1 * * ^ ' . \ " ■ - * * 

for ,t)ie great majority of stud^ntsr The ex&nt to which *students and'", t 

schflol^i^ouljd or .9ouia take advantage of this y^^^^^^^^^^^^^ it -pare avail- 
able is, not («;nowa, / * / " * 



ivr Factor Analysis ^ t- , ■ < 

Factor aiial^sis was used ^s a second method of examining the 
structure 'of *these tvo ^oriis of the GRE Psychology Test* The Psychology 
Committe'e* s content analysis is a logical way of relating test -contend 
to ^he curriculum, and^is intended" to insure that 4II the tjajoij'areas 
of curriculum are appropriately represented in the tests^ In view of the 
denonstr^tlLn of the ^relative independence of the subscorel bas^ on con- 
tent areas, the results of the content analysis, became relatively less Im^ 
portant and the fact that. each content area did not emerge a^ a separate 
factor was not disturbing, lioreover, such a result is consist*ent vith the 
fact that content areas are nbt learned in<}ependeiitly, that many required 
introductory courses cover all the major areas of psychology, ^nd that much 
psychological theory is applicable across content areas* ^ ^ • 

Factor analyses offers an independent:, parsimonious description of 
^ the Observed data* This is a legitimate way to study the structur^or 
the tests, and the results should offer inslghtf's about the way the t^stlf 
are fOnctioning. " ^ 



Factor analytic techniques vere us6d '^(3 .iWescigat^ cvo i^eparace 

questions: - ^ ' ' [ ^ , *, * 

!• To what extent do the icems in ea^h o-f'Che experlnental, ^ub- J.' 

' scores ^appear to "measuring a single general, factor^ ■ ; 

2. Do othe^ logically -valid and potentially useful groupings «^ 

(subscores) of the items eScist in the data? ^ 

Theoretically, one wuld proceed by /Oljtalning the interjcorrelations 

among all- 200, qvestiojra for each of the:^ests* In practice^ xhis number 

of variables (questions) is far toa^mariy to econodicaliy- subject j:o ai 

conventional factor atltelysis- Therefore, each' test was split ^'into two 

parts, 'the first consisting of the odd numbered questions and the -second 

•or the even numbered questions^ The basi<i idea behind this approach is 

tn. ^ntui^ive o^ae . ^ Id is esscned that the odd numbered Iteas of a lozg 

ztsz *^'p\:li have the sane number of factors as the evea ntaabered itiens 

01 the-cest, a:!d uthat these t%^^ sets of factors %*ould be approximations 
^ • 

of the canqoa factory of .the tota'la^'t^st.' Furthermore, it niay be e:cpected 

canc:ilcal corr3la54.ons J>etveea the principal component ^^^ ^o^ 
tr;t odd ^iL^berad^^^^s and the even r-uaberad ite^ would ir-dica:e bjf 
their- jzagnitudfl^S^fSfe number of izportanc factors in each subset of iCs:::s 
'and 90 of the whole test, provided that enough principal cooponants were 
canonlcally correlated for some small canonical correlations^ acong'thefn 
to indicate that there were. ur^imDortant components that could be set 

;aside as random error, 

if 1 

A codoor.ent analysis was obcaLied for sach part, consistlr.j, of 9i:c 
principal corspor.ears* The correlations of the six principal coa?oner,ts 
of the * two par:s of each test were obtained, and subjected to canonical 



^correlation analysis. In each case, this analysis revealed" one very 
highly corralatad pair of canoait;^! vvariat££^j Tde caree siaallest canorfi- 
cal correlatiocil *jere distinctly, trivial iu each case. 

The six canonical variaces were condensed into three orthbgonal 
"<juasi-f actors'* by rescaling to unit variance the sum- of 'each pair of 
correlated canonical variates. The$a three siaas of corresponding odd 
and even canonical variates vere correlated with the original itea scares, 
r^sf rascaled s:ns of corr^saohding c'anoni-cal variates are aporoi^i^tions 
to ths principal coaponant scores of the sanole matrix of correlations 
acong all 200 itsms. TJieir correlations with the original item scores- 
is therefore an appro:cimation to the principal cofliponfent, analysis of all^""' 
200 itsos into three components , It was this aati^ix of correlations of 
itea scores Cw'iti'apprpxinate 'principal comoonent, Scores) thatTwas treated 
as if it were a factor matrix and transformed to ,an^ obliqxie solution- 

The canonical correlations J)^cween the six odd numbered and silt - - ^- 

* ■ - . 

evazf numbered Iten co:iponent3 are presented in Table 6. Xn each 

Case fom onXj one of the correlations exceed? the .80 standard suggested 

In ihe proposal as appropriate to identify factors corcon to both sets 

of caca. Tnis ?ujgest3 that if one were seeking the aost parsjja^ious ✓ ^ 

c*jt adequate description of the obse^ed data^ each test could be 

described essentially in terms of a single factor, Xhls. description of 

the structure pf each test is* of course* consistent with .the practice 

of having a single CSZ Advanced Test > in the area of psychology and with 

reporting a total score for the test. « 
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Table & 



Canonical Corr^latioas Bexveek Sis 
Six Odd Numbered' Item Components and Six 
Even Numbered 'Item Comoonents 





VG%2 




1 


.8931 


.8933 


2 


.5091 


.4985 


, 3 j .'3782 


.4130 

1 




,2125 


.1925 


5 


.1420 


.1753 


6 1 , 0499 


ai74 ' 



Iz acdlriDr, ro the one well defined ficiot in each test Sora, ther^ 

ara t\*o less clear!/ dsfi^ad factors; che cnrae rema^ining q^nonicsl corra, 

laciOM in each for:^ are definitely trivial., Bscause the purpose of this 

analysis was to seek potentially useful groupings (s^ibscores) "of the 

ru^3:ior-3> ve cor.sldarad in de;:all tha chr^a faccor solution, 

* 

The finding that the encire test contains oaly one predomi::ant 
factor suggests that it is unnecessary to subject each of the pontent 
area subscoresto factor analysis proposed for the research project. 
The results of the iceibt analysis demonstrated chat^ with the exception 
of a few items, the subscores fcire scales that "hang together" rather w&ll. 
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Tables of*the fifty items wltH^.the highest congelations^ with each 
of the three factors in the obligufe fsftucture matrices ar6 presented in 
Appendices IVa and IVb; The ^r^iations amonjs tha factQrs in each form 
are presented Iji Table 7 below;' /Although the factors have substantial 

correlations with each other^; they are less highly correlated than ^he , 

' ' - 

subscores being reported at present. 

* ^ i - ^ - ' ^ K ^ - * 

Table 7 

Correlations^ Among the Three Factors 
in the Oblique Structure Matrices 







K-UGR2 


— ^ r 


Factor 


I 


II 


III - 






.523 


.652 


II 


,523 




.SO'4 . 


III-'* 




.504 








~ K-VGRl 




Factor 


I 


II 


III 


I 


• 


.548 


,602 


II'* 
III 


,548 
.602 


.561 


,561 



^SKt escaained the factor analytic ^results in relation to the 
consent classifications of the items.. Table 8 gives the content 

.class if icatio-rf^s of che fifty items with tne t^ignesC correlations with 

I 

each of the three factors in VGRl. Of the 50 items wich highest corre- 
Z£Cion5 vita factor 1, of them contribute to che Er^perinental ^su1>- 
score '(cci^eqrc araas 2,6,7) and only 2 to the Social Psychology subscore 
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(;:cnrsz^£r3as l,C,5,i>. C: the 50 ir&ns 'Chi hi^'jias: corr^lzcions 

vizh zacsor 2, 23 of chart coKribute lo-ihe Zipper lizeciral subscora, and , 

23 CO tn^ Sociel subscore. Of che fifty i-taas wlch the "highasc ct>rrela* 

tioas vltb: fawo'r' i, 22^ C9:i;ribuce co the E^n^arizi^ncal subscor^e and 23 

to^tha Social subscore. Faccor 1, than, -is prttdoaina^itly in che area ^ 

oi Exp^ri^aaral Psychology, while factors" t aad 3 are abojtit even^ 

cividac bsl^'as;^ che ar^as of Ex?erinar,cal aai Social ?syciiol<^y. ' 

• * \ 

Table S ^ ^ . 

Cocicanc Cl«sif icacioti of -^he SO vich che 

Highest Correlations on Sach of che Three Factors in VGRl 





Factor 1 


» s 

Taccor 2 


Factor 3 ! 

\ 


1* Terscnalitj 


1 ^ " 


. 10 




; 2. ^ Learning 

* 


11 


13.. ^ 


life J 


! 3* Maas^Iren^nt 


4 


■'-3- 


5 

- 




1 


4 




2 ' 3 ; 


; 6. _?hy^siolcji^i S Coa^araci^e 






\ 7. ^tetck^zzal i Sensory 






' ! 


' . Clinical i Abnonadl 


- j 


10 


6 '{ 


! 5f. Appiiad ^ pcher^ 




I - 


- ! 

i 
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:3 :o7^ o- Cne uare ore- 



r^a Co^znittae idsncizied fai:Cor 1 as a??sarir.s co caasura kr^ovisdgs or 

?s7chological facts, such as che functions of rods aad-cones, prlnarily 

in coaceat areas 2, 6, and 7. The Cozziii^ee identified faitor 2 as 

a^^^aping to censure knowled^ge o£ psychological theories across all the- 

ciacaat areas. A typical ic^ vould be: "The structural concepts of 

' ■* 

trlkson's theory are referred co as*..." The^ Cocaitcee identified 

faccor 3 as orii^rlly/ieasurir.g cha candidates^ pothers o: analysis and 

> ' • f 
i:::er?recation, It^as vith hijh corrslanioas wich^faccor 3 typically 

J 

re^uirad che candidate to analyze or internet stimulus i;:ate3?£al pre- 

* 

sented in che test* 

The Co^aictee^did not believe that subscoMs based on che^e factors 

voiild be useful for guidance and placement .\ The point was aade that the 

curricul>3Zi is organised a^ong contant-ceterained lilies, and chat st:idents 

■ ' .1 

uaed to 'iasow their screagchs and veaSaessas in chose tens. The Cozsiccse 
the results of che factor analysis in no^ vay linited the useful^ 
ness of subsc^^res based on content areas. 



table 9 on che following page gives che 'Conceric c 



cacio;is of 




ir Jami* 



the 50 icenis.^ith the highest correlations ulch each of the three factors 
in 0GR2, which were discussed with the Psychology Committee^ 
ary 1976 meeting* Of the SO items with the highest correlatioiiFwith xac* 
tor 1, 21 of them contribute co t^je Experimental Psychology subacore, and 
17 to the Social Psychology aubscore. Of the SO items with the highest 
correlations -with factor 2, ox chem contrituCe to che Expeciaencal ?sy* 
chology aubscore, ^nd only^^A to che So(;ial> Psych^jlogy subscore* Of che ^0 

i 

it^ms with che highest: correlaciona, vith factor 3, 3^ or chem concribuce 
to^the Experimental Paychology subscore and only 3 to the Social Psychology^ 
Subscore* > 



\ 
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Table ^ 

Content Classification 'of the 50 Itens with Che 
Highest Cprrelatioos on Each of the Three Factors in 



t 


Factor 1 


Factor 2 




1» , personality ^ 


S 


• 2 


1 


yz. Learr.ins 


" 11 


- 9 


- » '1 


3* Measurezent 




4 


11 , i 


D^vslcpz^tLtal ' . i 3 * 


1 


i 

■ 0 i 


5" Social 


5 


0 


3 


1* 6* Physiological ^ Comparative j 3 


17 


3 


j 77 Percapt-jal h Sensory j 5_ 


1 ■ " 




j ' 8. Clinical i Abaoraal 


■ 


0 


1 4 


; 9. A^olied"! 0:her " ) ' 1 

i ' 


0 


0 



Although there ^e some obvioua differences between these results and^ 
those obtained'for VGRl in tenos.of the content classifications of the 
items, the Conmittee felrt that the two analyses are basically consistent 
in identifying fac;;ors which a&asure priaarily knowledge of facts, know-* 
ledge of theories, and powers o£ interpretation and analysis- In this anal- 
ysis, the first factor is aLnost* evenly divided between the Esfperiaental ^nd 
^Social Psychology areas > while factors 2 and-3 *are made predominantly of 
items in the area of Expetimental Psychology* In form VGRl, the first factor 
correiaJted highly with items in the Experimental area, while factors 2 and 
3 were about evenly divided between the Experinental an<l Social Psychology 
*areas*^ ^ - ' 



It^appaars from Tabled 8^ aad 9 tha^. the first factor jLn this-VGRl 
analysis, which correlated hlghKyj&ith .Iteipa In the areas of Physio- . ^ 

All / " ■ ' ■ ■•■ : ■ > 

logical and Comparative Psycholomj Perception 'and Sensory Esychology, and ,_ 
Learning, corresponds to the sewfc^ factor In Content analysis at ,7^ 



the item level coofirms this, t^s^otj^Z In UGR2 correlates highly with 
items that tap the same kind oj^^ovledge of facts in EKperimentar Psy- 



chology as those in fa^tbr i of ^GRl* ^ 

Of ^he seven it^d^ wit^ thr^ighest^cprrelations on factor 1 in UGR2, 

■ ft ■ 1 9 ■ • '■ • ' 

six are in the area^^tneastirement* This factor is characterizedtby items 
reji^iring a knowlepn^^f tijs theories of ^p'^ychology, ' including knowledge , 
of measuife theory^} /dnd: corresponds to the second factor in-VGRl* ■ 

As in the ^a-i^j^s^fe of VGRl, the*^ items which correlated highly with the 
third factor 1^ end Jto those which require the candidate 'to analyze 

or interpret' sti^l^ material presented^ in the test* * 

/ \-/ ' 

For purposes of this study, it appears that t;he similarities between 
the -factor analyses of the two forms are most Important* The analyses pro- 
vide th^e Committee of Examiners with a new way of looking at the structure 
of th^ test. While subscores based on th'ese factors are logically defensi- 
hie, the Committee of Examiners does not believe they would be useful fof^ 
purposes of guidance aa4 placement* Because of the Committee^s reservations 
about, s'^bscores based on the results of the factor analysis, these subsc^es 
were^tiot subjected to the reliability and independence analysis that were 
applied to the Committee's content-based^'subscor^s*' Likewise, no compari-^ 
son .i?as made of how well the Committee's content analysis fits the data in 
coi^arlson with the structure determined by £a<^or analysis. 
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.Conclusion ^ , , * # ' 

TJie goal of chls study haa been co see If a sec of sltbscoresi, which 
would'be both meaningful and stiiclstlcally appropriate 'for guidance and . 
placement declalons coul'd iJe identified, whlle^maincalnlng Che ablHtyof 

4 ' • ' 

che Psychology Test co.,produce*'a single scote with appropriate -content 
coverage and staclsclcal ^jprop^rtles for use In adSl$slond decisions^ 

^ ^Subacores based on -che content areas def Ine^^y^ che CCT&lcce^ appear 
CO have' substantial pot,enclal for providing useful information about stu- 
dents wlch unusually high or low scores for purposes b£ guidance and 
placeigenc. Comparisons between pairs of subscores would allow judgmencs 
that some subscores are higher than others -co be made for most students 
with 80 to 85 percent accuracy* . . 

Finally, the factor analysis showed only one well-defined factor In 
eaeh of the test;3, which can be described basically in tetms of a sliigle 
factor^ Subscores based on the three-factor solution <f6 not appear to , 

'have much potential for us& an guidance and placement decisions ^ 
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^* t* ^Ic^ Statistics, by Subscore: - Sununary of Keans and Standard Devia* 
tions of De^c^s and Biseridl Corr^lacions 

' ' ■■ " - * 

A* For Form (10 tables) * " 

; ■ [ ... * 

B* For Form K-VGRl (10 .tables) ' , ^ ^ 

II'* Correlation Tables , ' * ^^ - h 

^ ^ . A* . For F4fin. k-UGR2 

For Form K-VGRl 

C* 'Summary cf Item OvJ^^ia^^ j^acfa g Variabifes 

III* Citation from Lord, F*M*, The Utt&2ati6n of Unreliable Dlliereoge 
^Scores, Edxicational Testing Service, RB-58-1, 1958* * ' 

IV* ^ Item/Factor" Correlations in ilamc Order, for each Factor and for 

Cotanunality * " ' 

A*, For. Form 1::-aGR2 (4, tables) : 

B* For Form K-VGRl '(4 tables) 
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.Social Psychology subscore. . , * 
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.Jn Lord^ s theory'a difference is treated as a real difference only If the absolute value 
of the difference Is greater than the standard error of test x plus the standard error of test y. 
As an approximation It Is assumed here that the two subtests have equal standard errors^of 
measureoaent* According to Lord (1958) "moderate ^deviations from this assumption will 
cause only minor differences in the results obtaUied. 

A difference, d# Is to be tjreated as a real difference only i^ « 
(2) * I d 1 1 2 S,E. . ^ ' ' ' . 



^ere I d 1 represents the absolute value of the difference. Since the errors of 
measurement in tests x and y are presumably Independent, 



(3) 



S.E. ! /sE ^-+SE ^=%r2"SE 
d X y X 



By the usual formula for any standard error of measurement SE ^ a, V 1~t,.; 

c u ud 

or, expressing d in standard measure so that d~ I, ^ 

It Is seen from (2), (3), and (-1) tm the difference will be treated as a real difference only If 
<5) • I d t i V 2( I'T^^) . . ^ > , 

The cattiiig point given by the right- side of ( 5 ) will be denoted by K. ' - 



Consider now the bivariate frequency distribution between the observed difference, d, ^and the 



true difference, A * It is well known that the correlation between d and A is equal to / r 
It win be assumed here that the bivariate distribution is normal* 



dd* 



The proportion of cases lying above d = +K in the bivariate distribution is the proportion 
of examinees i6x whom tbe judgment A > 0 is made. The proportion of these that actually 
lie above A = 0 is the proportion pf judgments correctly made. Similar statements hold for 
the Judgment A < 0* 

Once the reliability of the difference score is known* these proportions are readily 
determined from tables of the bivariate normal frequency distribution* '* 

' .Some illustrative results are given in Table 4, on page 10 of the text* The last two columns 
"represent Ijoth Judgments of A > 0 and Judgments of A < 0. ' , - 
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